
LEARNING POINT 



North Central Regional 
Educational Laboratory 



EDUCATIONAL 

TECHNOLOGY 

NEWS 



to the sixth edition of Educational Technology News, a biannu- 
al newsletter published by the Center for Technology at the 
North Central Regional Educational Laboratory (NCREL), a 
wholly owned subsidiary of Learning Point Associates. As 
one of 10 regional educational laboratories funded by the 
US. Department of Education, NCREL is a leading research 
laboratory, and its designated National Leadership Area is 
technology. In partnership with the North Central 
Eisenhower Mathematics and Science Consortium (NCEM- 
SC) and the North Central Regional Technology in 
Education Consortium (NCRTEC), we continue to enable 
hundreds of teachers, administrators, and policymakers to 
integrate technology effectively. 
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In this issue, we present a look at recent Center for Technology research and development efforts. In particular, we've included 
the following two articles: a summative, in-depth look at the research and findings of A Meta-Analysis of the Effectiveness of 
Teaching and Learning With Technology on Student Outcomes (full report available online-see article for details) and an informative 
peek at the content of NCREL's Quick Key No. 7 about scientifically based research. Also appearing in this issue is an announce- 
ment about presentation materials from the 2004 National Educational Technology Conference held in conjunction with the 
2004 NCREL Annual Conference March 10-11. 



We thank you for your continued interest in our work and invite you to learn more about these and other Center 
for Technology research and resources by visiting our home page at www.ncrel.org/tech/. For more information 
about Learning Point Associates, please visit www.learningpt.org. 
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Further Research Suggests Classroom 
Technology Use Has Positive Impact on 
Student Performance 



By Robert Blomeyer and Rosalie Guerrero, NCREL Center for Technology 



A S states and school districts are increasingly 
required to choose evidence-based programs 
for their schools and classrooms to meet the 
scientifically based research requirements of the No 
Child Left Behind Act (2002), educational research 
must provide sound empirical evidence that pro- 
grams and interventions are improving student aca- 
demic outcomes. To address this need, NCREL com- 
missioned a synthesis of scientifically based research 
on the impact of technology in teaching and learning 
on student outcomes. 

The report from that synthesis, conducted by Hersh 
Waxman, Michael Connell, and Jon Gray (2002), 
examined 20 high-quality educational research 
studies on classroom instruction with technology. The 
researchers found evidence supporting a small, posi- 
tive effect (.31) of teaching and learning with technol- 
ogy on students' cognitive, affective, and behavioral 
outcomes (Waxman et al., 2002). Given the small 
effect observed and the low number of studies high- 
lighted in the report, NCREL continued its work with 
Dr. Waxman in 2003. 

This work, A Meta-Analysis of the Effectiveness of 
Teaching and Learning With Technology on Student 
Outcomes (Waxman, Lin, & Michko, 2003), examines 
again the effects of teaching and learning with tech- 
nology on students' cognitive, affective, and behav- 
ioral outcomes but with a larger sample. In this 
research report, 22 more high-quality studies were 
added to the 20 studies from 2002, resulting in a sam- 
ple of 42 high-quality educational studies on class- 
room instruction with technology. 

Analysis of the research contained in the 42-source 
synthesis shows a small, positive increase in the effect 
size from .31 to .40 (Waxman et al., 2003). This 
increased evidence of effect justifies greater confi- 
dence in the conclusion that classroom-level use of 
technology has a generally positive effect on stu- 
dents' cognitive, affective, and behavioral academic 
outcomes (Cohen, 1988). 



While numerous meta-analyses have examined the 
effects of technology on student outcomes (typically 
achievement)— with several recent analyses focusing 
on the effectiveness of specific technologies— few 
have examined how technology is appropriately 
integrated into classroom instruction. In recent years, 
the rapid improvement in technology suggests that 
technology use examined in the past may look 
dramatically different today. The Waxman et al. (2003) 
study begins to address these issues by examining 
the effect of classroom-level technology use on stu- 
dent academic outcomes. 

Purpose 

The ability to examine differential effects of a treat- 
ment or intervention is one of the many advantages of 
using meta-analysis to aggregate and report educa- 
tional findings (Hunter, Schmidt, & j ackson, 1982). As 
the Waxman et al. (2003) study continued the work 
begun in 2002, its purpose was to synthesize recent 
research on the effects of teaching and learning with 
technology on student outcomes. The researchers set 
out to answer the following questions: 

• How extensive is the empirical evidence on the rela- 
tionship between teaching and learning with tech- 
nology and student outcomes? 

• What is the magnitude and direction of the relation- 
ship between teaching and learning with technolo- 
gy and student outcomes? 

• Are there certain social contexts or student charac- 
teristics that affect the relationship? 

• Are there particular methodological characteristics 
that affect the relationship? 

• Are there specific characteristics of the technology 
that affect its relationship with student outcomes? 

• Are there specific characteristics of instructional fea- 
tures that affect technology's relationship with stu- 
dent outcomes? (Waxman et al., 2003, p. 6) 
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Method 

From the 42 studies analyzed, Waxman et al. (2003) 
identified and coded common core characteristics of 
technology, teaching, and learning (see Table 1 on 
page 4). The common characteristics of technology 
were adapted from previous meta-analyses in this 
area, while the teaching and instructional characteris- 
tics were adapted from The Five Standards for 
Effective Pedagogy developed by the Center for 
Research on Education, Diversity, and Excellence 
(2002). (The list of coding categories used in this study 
can be viewed at www.ncrel.org/tech/effects2/ 
appendx.htm.) 

Findings 

Approximately 40 percent of the studies focused on 
elementary school (Grades K-5), 40 percent on middle 
school (Grades 6-8), and 20 percent on secondary 
school (Grades 9-12). Thirty percent of the meta-analy- 
sis studies identified schools that use personal comput- 
ers, 26 percent that use networked laboratories, and 5 
percent that use multimedia. The other 39 percent iden- 
tified schools that use a variety of other technology 
resources. The most commonly reported (31 percent) 
uses of instructional software were in an exploratory 
environment such as simulations, hypermedia, and 
hypertext, with a few using drill-and-practice software or 
other productivity tools. 

The study looked at three types of student outcomes: 
cognitive, affective, and behavioral. The most com- 
monly reported cognitive outcomes were researcher- 
based tests, authentic assessments, and standardized 
tests. The majority of affective outcomes were student 
attitudes about computers, followed by students' moti- 
vation or self-concept. Behavioral outcomes were most 
frequently reported as the number of tasks attempted, 
followed by time students spent on task and persever- 
ance. Overall, the mean study-weighted effect sizes 
averaging across all outcomes was .41 (p < .001), with 
a 95-percent confidence interval of .17 to .64 (Waxman 
et al., 2003). According to the researchers, "This result 
indicates that teaching and learning with technology 
has a small, positive, significant (p < .001) effect on stu- 
dent outcomes when compared to traditional instruc- 
tion” (Waxman et al., 2003, p. 11). 

Discussion 

Waxman et al. (2003) went to great lengths to ensure 
that the meta-analysis included high-quality research 
studies; however, the researchers point out several 
limitations of this study. Research quality was a great 
concern, as few examples of sound research were 
found suitable for this study. Many studies were not 
included on the basis that they had reported data that 



did not permit the calculation of effect sizes, they had 
not used an experimental design, or they had neg- 
lected to report sufficient details to allow adequate 
coding of study characteristics. 

Other limitations of the meta-analysis are related to the 
study design. First, due to the correlational nature of 
meta-analysis results, causal inferences based on these 
results are not recommended. Second, meta-analyses 
do not have any control on original data; subsequent 
analyses are limited by the quality of the primary stud- 
ies. Third, only refereed journal articles were included; 
other research available via Web sites, books, chapters, 
dissertations, conference proceedings, and technical 
reports was excluded because of quality concerns. 
Finally, this work is limited by the relevance of the 
research analyzed: A few of the studies reviewed took 
place in the early '90s; therefore, the technology used 
in the earlier studies is now at least a decade old. 

Conclusions 

The results of this analysis indicate that overall the 
effects of teaching and learning with technology on stu- 
dent outcomes are at least twice as large as, and may 
be greater than, reported in the 2002 meta-analysis 
conducted on instructional technology. Flowever, the 
researchers maintain that there are still questions 
regarding the effects of teaching and learning with tech- 
nology on student outcomes (Waxman et al., 2003). 

In attending to these important questions (as they con- 
cern the impact of classroom-level technology use on 
student academic outcomes), we need additional 
high-quality educational research that addresses the 
issues and questions defined by the Instructional/ 
Teaching Characteristics and the Technology 
Characteristics (see Table 1). It isn't enough to simply 
claim that "more research" on technology and learn- 
ing can or will make a difference. Additional experi- 
mental and quasi-experimental research studies are 
needed that adhere to the highest standards for qual- 
ity and address the critical common core of technolo- 
gy and instructional factors defined by the Waxman 
protocol (explained in the report's Method section 
[available at www.ncrel.org/tech/effects2/method 
.htm], which covers the review criteria and research 
synthesis techniques used in the meta-analysis). 

A Meta-Analysis of the Effectiveness of Teaching and 
Learning With Technology on Student Outcomes is 
available on the Center for Technology's Technology in 
Education Web site at www.ncrel.org/tech/effects2/. 
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Table 1 . Common Core Characteristics of Instruction/Teaching and Technology in the Classroom* 

Instructional/Teaching Characteristics 



• Joint Productive Activities - coiiaborative instructionai activities required and positiveiy supported 

• Language and Literacy Development - student ianguage and iiteracy connected to content-area knowiedge 

• Contextualization/Making Meaning - using reai-worid exampies to express understanding 

• Challenging Activities - advancing students' understanding to more compiex ieveis 

• Instructional Conversation - frequent or reguiar faciiitation of conversations regarding students' views, which 
are supported by resources 

• Setting - ciassroom, networked ciassroom, or computer iaboratory 

• Mode of Instruction - whoie-group to individuai 

• Role of Teacher - iecturer, faciiitator, modeiing, or mixed 

• Teacher Qualifications - aiternativeiy certified, certified in content area, not certified 



Technoiogy Characteristics 



• Type of Technology - type of hardware and network avaiiabiiity 

• Software - tutoriais, driii and practice, expioratory/simuiations, productivity toois, programming 

• Technology Resources and Support Availability - minimai to ampie 

• Role/Focus of Technology - purpose of use: unspecified, productivity, deiivery system, or resource 

• Quantity of Technology - iittie (fewer than three) to ampie (more than nine per room) 

• Number of Computer Sessions - the reported number 

• Duration of Computer Sessions - average number of minutes per session 

• Teachers' Experience with Technology - unspecified to very experienced 

• Students' Experience with Technology - unspecified to very experienced 

• Teacher Training in Technology - number of hours of training 

• Feedback and Assessment Practices - no feedback, minimai, or eiaborate 

• Learning Responsibility - student-, teacher-, or system-directed; or mixed 

• Task Difficulty - difficuit, moderateiy difficuit, not difficuit, or mixed 

• Type of Learning Task - basic skiiis, probiem soiving, inquiry, or mixed 

• Type of Technology Program - basic skiiis, probiem soiving, inquiry, or mixed 

• Pattern of Student Computer Use - teacher use, presentation station, number of students per computers 

• Percentage of Student Computer Use - percentage reported 

• Objectives of Computer Use - remediation, writing, communication, research, anaiysis, presentation, 
coiiaborative, or independent work 

• Skills were identified based on the 42 studies included in the 2003 meta-analysis. 
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NCREL Quick Key No. 7: 

A Foundation for Understanding 
and Evaluating Scientifically 
Based Research 



New in the 
Center for Technology 



By Matt Dawson, NCREL Center for Technology 

O ne of the underlying principles of the No 
Child Left Behind (NCLB) Act of 2001 is that 
school reform efforts to improve student aca- 
demic achievement must be based on "scientifically 
based research” (NCLB, 2002). This phrase, while 
seemingly innocuous, has several implications for 
school leaders as well as teachers. 

With this fundamental NCLB principle in mind. 
Learning Point Associates has published Quick Key 
No. 7, "Understanding the No Child Left Behind Act 
of 2001: Scientifically Based Research." It was devel- 
oped to provide a brief overview of what scientifically 
based research means for educators and administra- 
tors by demystifying the three types of research: the- 
oretical, correlational, and causal. This latest Quick 
Key covers two main topics: (l)the definition of scien- 
tifically based research, including practical examples 
to help demonstrate the concepts; and (2.) a general 
discussion of research, as it provides a basis for under- 
standing and evaluating research claims. 

Quick Key No. 7 joins the series of previous topics 
assembled to help educators and administrators 
respond to the NCLB Act: 

• "Understanding the No Child Left Behind Act of 
2001: Reading" (Quick Key No. 1) 

• "Understanding the No Child Left Behind Act of 
2001: Opportunities for Schools in Need of 
Improvement" (Quick Key No. 2) 

• "Understanding the No Child Left Behind Act of 
2001: Technology Integration" (Quick Key No. 3) 

• "Understanding the No Child Left Behind Act of 
2001: Mathematics and Science" (Quick Key No. 4) 

• "Understanding the No Child Left Behind Act of 
2001: English Proficiency" (Quick Key No. 5) 

• "Understanding the No Child Left Behind Act of 
2001: Teacher Quality" (Qu/dc Key N o . 6) 

Quick Key No. 7 serves as a foundation built to pro- 
vide a clear understanding of what scientifically based 
research is and what it is not. In particular, the NCLB 
Act (2002) lays out six components of scientifically 
based research: 



• Empirical methods are used to carry out the research, 
which is conducted in a systematic and consistent 
manner, with keen attention to detail. 

• Data collection and analysis are rigorously conducted 
to ensure that the data are collected, analyzed, and 
interpreted correctly. 

• Measurements or observational methods that provide 
scientifically valid and reliable measurements across 
many different measurement points and observations 
are used. 

• The studies employ experimental orquasi-experimen- 
tal methodology to optimize the researchers' ability to 
answer the questions under investigation. 

• Enough data and description should be provided so 
that future researchers can attempt to replicate the 
findings by conducting a study using the same meth- 
ods and instruments. 

• An independent, objective, and rigorous external 
review of the research has taken place. 

These requirements are important to remember, in 
that while there may be many studies that show some 
sort of impact, only studies that meet the six criteria 
above will be considered scientifically based research 
according to the tenets of the NCLB legislation. 
Knowing the principle requirements of the NCLB Act 
is a necessary early step in understanding and evalu- 
ating scientifically based research. With these six 
requirements in mind, one can better evaluate the 
various kinds of research and how certain research 
may connect to practice. 

In order to meet the criteria set forth in the NCLB leg- 
islation, school leaders must find ample research- 
based evidence for their reform program of choice. 
Finding scientifically based research and building a 
cumulative research base is certainly a daunting task, 
but Quick Key No. 7 is a resource developed to help 
school leaders understand, evaluate, and conduct sci- 
entifically based research. 

Quick Key No. 7 is available on the Learning Point 
Associates Web site at www.ncrel.org/csri/tools/ 
qkey7/index.html. To order print copies of this or other 
resources, visit the Learning Point Associates Product 
Catalog at www.learningpt.org/catalog.htm. 
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Making Good Choices in Education: 
More Than Just the 2004 
NCREL Conference Theme 



In the Spotlight 



By Nicole Gallmann, NCREL Center for Technology 

T he 2004 National Educational Technology 
Conference, held in conjunction with the 2004 
NCREL Annual Conference on March 10-11 in 
Naperville, Illinois, was attended by a record number 
of participants. In addition to its guidance as a confer- 
ence theme, "Making Good Choices in Education” 
gave voice to the drive and determination of so many 
educators who approach educational technology 
concerns each day with innovative strategies and 
tools, prompting those attending the conference to 
join in the conversation. The dedication of this year’s 
conference presenters and participants will continue 



to shape and further apprise the work of the Center for 
Technology, as well as that of countless practitioners, 
policymakers, researchers, and other stakeholders, 
about making not only good, but informed, choices 
in education. 

For panel and breakout session presentations, hand- 
outs, abstracts, presenter and participant lists, and 
archived conference summaries, please visit the 
National Educational Technology Conference Web site 
atwww.ncrel.org/tech/netc/. We look forward to see- 
ing you at next year's conference scheduled for March 
9-10, 2005. I 
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